Image processing for use in a still or video camera consists of image reconstruction, encoding and real-time camera control. Image reconstruction is typically divided into pre-processing, main processing and post-processing stages, as shown in FIG. 1. Additionally, a memory module can be introduced between the pre-processing block and the main processing block, as shown in FIG. 2. Additional pre-processing can also be made by SW as a memory-to-memory operation.
The role of pre-processing is to normalize different sensors for the main processing stage. The real-time functions are used to adapt to the changing conditions in imaging scene. Real-time functions such as automatic exposure need (statistical) information from the image. Other real time functions include automatic white balance and auto focus. The statistical information from the image is normally collected with a hardware statistics block. Real-time functions are typically carried out by a software block in order to provide various parameters to the main processing. The real-time functions block may also provide various parameters or controls to the image sensor directly or through the main processing.
Pre-processing contains a variety of processing steps that are needed to change the original raw image captured by the camera sensor to a form that can be used in the main processing or in the statistics collection stage. The output of the pre-processing stage is still in raw format. Examples of pre-processing include shading compensation, dead pixel correction, dark current elimination, pixel linearization, noise reduction and green imbalance filtering. These pre-processing steps are carried out either in the camera sensor or in a separate block in the image processor before the main processing stage or the statistics collection stage. In the processing of a raw image, pre-processing is run only once. Image statistics are collected from the raw image for imaging control purposes. Among the typical imaging controls, automatic exposure control, automatic focus control and automatic white balance control require some statistics in the analysis and synthesis part of the algorithm. Other controls include automatic brightness control and automatic contrast control. The statistics can be collected in the forms of histograms, block averages high-pass and band-pass filter sums. Main processing typically contains color filter array interpolation (CFAI) and white balance (WB). All the processing steps that are carried out before WB and CFAI are usually referred to as the pre-processing steps. Main processing changes the raw image to a form that can be shown to an observer. Image display and print are two of such forms. Main processing may contain other operations such as color space conversion, non-linearization (gamma correction), image sharpening, geometric distortion correction, chroma suppression, dithering, image scaling. Most of the functions in these operations can also be referred to as the post-processing functions. Post-processing may also include image cropping. Image cropping as well as image scaling can also be implemented as pre-processing functions. It is possible to run post-processing functions with HW or with SW.
As the size of the pixels in a camera is reduced further and further, additional pre-processing steps may be needed. It is advantageous and desirable to provide a different imaging architecture to fulfil the present need in image pre-processing and to anticipate the future need.